Skip to content

fix: don't spend retry budget on undeliverable dispatch#95

Open
tjluyao wants to merge 1 commit into
mainfrom
fix/dispatch-undeliverable-no-retry-budget
Open

fix: don't spend retry budget on undeliverable dispatch#95
tjluyao wants to merge 1 commit into
mainfrom
fix/dispatch-undeliverable-no-retry-budget

Conversation

@tjluyao

@tjluyao tjluyao commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Problem

When the dispatcher selects a worker but the task cannot be delivered to it — the worker's node has no live subscriber on node:<id>:dispatch (receivers <= 0, logged as "No subscribers on tasks channel"), or publish_task raises — it requeued with _requeue_task(..., front=True), which defaults count_retry=True.

That spends the task's execution max_attempts on an infrastructure condition: the task burns all 3 attempts in ~3s and fails with error: max_attempts_exceeded and a null last_error, which reads like an executor failure and hides the real cause (an orphaned/stale worker registration — heartbeating but not consuming its dispatch channel).

Observed in production: an echo task died in 2.9s with attempts: 3, dispatched_ts: null, assigned_worker: null, last_error: null while the whole fleet was registered-but-dispatch-dead.

Fix

New _grace_then_fail_undeliverable() in dispatcher/base.py, keyed on a new TaskRecord.no_dispatch_since field (a separate clock, so the eligibility path resetting no_eligible_since each iteration can't clobber the timer). Both no_dispatch_subscriber and publish_failed route through it:

  • requeue without consuming the execution retry budget (count_retry=False),
  • after no_worker_grace_sec, fail with a descriptive error naming the orphaned worker + node,
  • no_dispatch_since is reset on a successful mark_dispatched.

This does not restore dispatch (that's an operational/field-side concern) — it makes the failure legible and stops infra faults from consuming retries.

Tests

  • tests/server/dispatcher/test_dispatch_undeliverable.py (+2): within-grace requeues with count_retry=False; post-grace fails with the descriptive message + reason/worker/node payload.
  • Full server suite: 435 passed.

When a worker is selected but the task can't be delivered to it (its node
has no live dispatch subscriber, or publish raises), the dispatcher requeued
with count_retry=True, exhausting the task's max_attempts in seconds and
failing with a null-error max_attempts_exceeded that masks the real cause.

Route no_dispatch_subscriber and publish_failed through a new
_grace_then_fail_undeliverable() keyed on a dedicated TaskRecord.no_dispatch_since
clock (so the eligibility path's no_eligible_since reset can't clobber it):
retry without consuming the execution budget, then fail after
no_worker_grace_sec with a descriptive error naming the orphaned worker+node.

Signed-off-by: Yao Lu <ylu@yao.lu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant